DOCUMENT RESUME 



ED 068 500 



TM 001 837 



AUTHOR 
TITLE 
PUB DATE 
NOTE 



Young, Jon I.; Young, Kent L. 

Model for Instructional Validation. . 



[72] 
1 3p. 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF-$0 . 65 HC-$ 3.29 

♦ Evaluation Methods; ^Instructional Materials; 
♦Measurement Techniques; Models; *Program 
Effectiveness; Reliability; Research Methodology; 
Technical Reports; *Validity 

♦ Instructional Validation 



ABSTRACT 

Two general areas of instructional validation are 
discussed: internal validation, which occurs within the framework of 
a single environment, and external validation, which involves the 
comparison of two different environmental frameworks. Several sample 
and basic designs are presented which may be used to validate 
instructional material. Two studies are then presented wherein data 
was collected to examine the usefulness of specific instructional 
procedures. Some comparisons are emphasized regarding internal and 
external validation procedures, although they both result in 
important data. Chief among these is the fact that internal 
validation occurs under strict controls, but the information acquired 
is only conclusive for the system in which the research occurs; 
external validity studies offer more generalizable results, but the 
conclusions cannot be as definite. For internal validation the test 
may be specifically designed to fit the system, but for external 
validation the test must be standard for both systems. A combination 
of designs may be used to validate instructional material both 
internally and externally in the same process in order to measure the 
lasting effect of the instruction. (Author/LH) 
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ABSTRACT 

This paper discusses two general areas of instructional validation and 
presents several sample and basic designs that may be used to validate in- 
structional material. Tinally, two studies are presented wherein data was 
collected to examine the usefulness of specific instructional procedures. 
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MODEL FOR INSTRUCTIONAL VALIDATION 



Jon I. Young Kent L. Young 

Brigham Young University 

Validation of instructional material must be effectuated in one of two 
ways. Internal validation is that which occurs within the framework of a single 
environment (i.e. , specific program, a classroom, a school). External valida- 
tion involves the comparison of two different programs, classrooms, or schools. 
Both types of validation have their strengths and weaknesses. 

Internal Validation 

The purpose of internal validation is to estimate the effectiveness of 
instructional material in terms of what the material is expected to teach. Internal 
validation limits the conclusions of the research to that setting wherein the in- 
struction occurred. If a set of instructional tapes are validated in & specific 
program, there can be no empirical way of claiming that the instruction will work 
in any other program. However, inference may logically be made that, In all like- 
lihood, the instruction will work in different programs and under different condi- 
tions . 

Despite the handicap of not being able to generalize results of research 
to other situations, internal validity studies offer some important advantages. 
First, these studies are mostly conducted under the auspices of strict control; 
therefore, conclusive statements can be made about the results. 

Second, internal validity is normally possible with a minimal amount of 
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time and effort. The instructional material and evaluation instrument tire 
normally available. The need to standardize an evaluation outside the immed- 
iate environment is unnecessary. Students to serve as subjects are readily 
available, and minimal disruption of their progress in the instructional system 
is maintained. 

Efficient internal validation studies require a minimum of fiva randomly 
assigned subjects per group. However, when possible, the total number of sub- 
jects in the entire study should be equal to or greater than thirty in order to 
obtain the maximum amount of information. It is equally important that all sub- 
jects be assigned to groups on a completely random basis, and that arbitrary 
assignment or purposive selection be used only as a last resort. 

Finally, internal validation should involve as small a segment of instruc- 
tion as possible and still retain meaning so that the results of student performances 
will- indicate specifically that the instruction is successful. If large amounts of 
instruction are tested and shown successful, additional studies are still needed 
to isolate the exact elements of the instruction which is producing the learning. 

A useful technique is to test large amounts of instruction and eliminate all the 
instruction that does not achieve the desired results. When some instruction is 
successful, the internal validation procedures are employed to isolete the best 
part of the instruction. 

Instructional Level Validation 

Instruction is normally designed to produce learning on a specified level 
(memory, classification, etc.; Gagne, 1965). To validate instruction for a speci- 
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fied level, students are presented the material and then evaluated at the speci- 
fied level, as well as other levels. For instance, if the instruction teaches a 
concept classification task, it would be expected that the student would not do 
as well on an evaluation aimed at another instructional level, such as memory 
or problem-solving. By testing at several levels the true instructional level may 
be determined . 

If naive subjects are not available, a pretest should be given. When a 
pretest is used, two groups are needed for each set of instruction. Design No. 1 
illustrates the model without a pretest and Design No. 2 with a pretest. 

Instructional Adequacy Validation 

This type of validation refers to the usefulness or effectiveness of the 
instruction. Design No. 3 illustrates the model for testing two sets of instruc- 
tion without a pretest. Another group is added for each additional set of instruc- 
tion tested. Design No. 4 illustrates the same model, but with a pretest. The 
additional group (a control) is used to determine how much learning is a function 
of the pretest. 

These two designs are possibly the most useful because they not only 
test the usefulness of existing instruction, but may be used to test potentially 
useful alternate instruction. Again, five subjects should be randomly assigned 
to each group. 

External Validation 



This type of validation procedure is more cumbersom to control, therefore 
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the results cannot be conclusive. Using external validation allows some con- 
clusions to be made concerning the usefulness of instruction outside the 
environment for which it was designed. However, specific recommendations 
are often not possible because of lack of strict control inherent in using large 
segments of an instructional system. 

External validation is normally conducted between entire classes, 
schools, or programs. The differences in their performances may be attributed, 
not only to different instructional material, but also to different teachers, more 
student involvement and other factors. 

A third problem with external validation is that measuring instruments 
are often created along the practical and philosophical lines of a system, and 
an evaluation specific to one system is inappropriate to measure another. Con- 
sequently, standardized independent tests are often needed to collect meaning- 
ful data. 

Finally, the basic assumption in testing two groups is that they were 
identical before the instruction and are identical in every other way. However, 
the fact that students select one class or program and not another indicates that 
they are not completely identical. It is not appropriate to assume that a group 
chosen from one class is the same as a group chosen from another class. They 
may have numerous differences in basic intelligence, motivation, or desire. 

Still, the usefulness of external validation is not to be overlooked. Sig- 
nificant amounts of a particular program or process can be identified as causing 
a trend for the better. Once this is established, the internal validation proced- 
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ures may be employed to determine which of the instructional materials is most 
essential and useful to learning. 

In this type of validation pretests are very important to eliminate subject 
differences so that only the instruction is validated. Designs No. 2 and No. 4 
are used for this validation, except a control group (Design #4) is used only in 
rare circumstances. A control group could be employed only if both groups (ex- 
perimental and control) were drawn from the same population of stud ants . The 
design is not valid if there is any basic difference between the experimental and 
the control. Students must be randomly drawn to fill each group, and the evalua- 
tion must be the same for each group. A combination of models could be used to 
validate instructional material internally and externally in the same process. 

Conclusions 

Although both internal and external validation procedures result in 
important data, some comparisons need to be emphasized. Internal validation 
occurs under strict controls, but the information acquired is only conclusive 
for the system wherein the research occurs. External validity studios offer more 
generalizable results, but the conclusions cannot be as definite. 

In all validation studies the critical prerequisite is an appropriate eval- 
uation instrument. For internal validation the test may be specifically designed 
to fit the system. For external validation the test must be standard for both 
systems. This requirement means that the test will be more general, and hence 
less information can be gleaned. Many studies finally turn to attitude tests, 
personality tests, anxiety tests and questionnaires. These types oi tests give 
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useful information on student attitudes, but in validating instruction, student 
ability is most important. 

The lasting effect of the instruction, either internally or externally, may 
be measured by administering the posttest (or a parallel form of the post test) 
some time after the first posttest was given. A combination of the presented 
designs may be used to validate instructional material both internally and exter- 



nally in the same process. 
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DATA COLLECTION 

Two internal validity studies were conducted in an individualized teacher 
training program (ISTEP) at Brigham Young University to illustrate some of the pro- 
cedures outlined in this paper. 

Study #1 

Two sets of instruction were being used in the program to teach students 
to correctly classify questions as memory, convergent, divergent, or evaluative. 
Design #4 was used because a pretest was needed to factor out any information 
prospective subjects would bring to the study and to use a control group to eval- 
uate what effect this pretest would have on a subsequent posttest. 

Method 

Thirty subjects were randomly selected from all the subjects in the pro- 
gram, These subjects were then randomly assigned to one of two experimental 
groups or the control group. 

All the subjects met together and took the pretest. Each subject was then 
given one of the sets of instruction or an irrelevant task. After the treatment the 
subjects took the posttest. The pretest and posttest were the same twenty-four 
item test. The test contained six items to measure each of the four categories 
of questions 
Results 

The scores on the posttest were analyzed with analysis of covariance 
using the pretest as a covariant. This analysis showed a significant difference 
between the groups (F=14.77, p<.001). The means of the treatment groups were 
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almost identical (19.872, 19.648), but the control group was significantly 
lower (15.479). 

Summary 

The results of this study show that either of the sets of instruction are 
useful in teaching the student to classify different types of questions. However, 
neither of the sets of instruction were completely adequate, as evidenced by the 
mean score of 19+ correct responses out of a possible 24. The result may also 
have been a function of the test. 

If only one of the sets of instruction is to be used it should be; selected 
because of its length. In this case the shorter set produced a slightly higher 
mean score. 

Finally, some indication is given for an appropriate criterion level of 
achievement. Since 19+ was the mean score, 19 would be a logical level of 
achievement. 

Study #2 

The study validated two types of instruction designed to teach the student 
to identify examples of the Modified General Model of Instruction. The first was 
a prepared film/tape and the second was a written discription. 

Method 

Twenty students had contracted to learn about the Modified General 
Model of Instruction. These twenty students were randomly assigned to four 
groups, three experimental groups and one control, using Design #3. This design 
was selected because none of the subjects had been previously involved with this 



Young, 1972 



9 



material, and were therefore considered naive, and it was not feasible to create 
a separate but parallel pretest in the time available. 

All the subjects met to receive their instructions. Group I was told to 
study the written material. Group II was instructed to study the film/tape pre- 
sentation, and Group III was instructed to study both the written material anci the 
film/tape presentation. Group IV was the control and the subjects were asked not 
to study for the test. 

The test consisted of twenty-eight items, and a criterion of achievement 
was set at twenty-two or more correct. 

i 

Results 

The results showed no subjects in the control group reaching criterion, 
while 60% (3 out of 5) of the subjects in Group III reached criterion and 20% (1 
out of 5) of the subjects in Groups I and II were successful. 

Summary 

Although no statistical analyses were made, the results indicate that both 
the film/tape presentation and the written description are most effective in helping 
students acquire this knowledge. Whether this is due to additional information in 
the two sets of instruction, or mere repetition, will have to be discovered by ex- 
perimental research 



DISCUSSION 

These two studies presented two alternate methods of estimating the 
validity of instructional material within a specific setting. The options for pro- 
cedures and types of design make a validation possible. The crucial character- 
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istics are an appropriate evaluation of the instruction and random assignment 
of subjects to the various groups. The only other concern is to filter out the 
previous knowledge of subjects. 

By following these basic guidelines, estimates of the validity ol instruc- 
tion are not only possible, but practical. 



VALIDATION DESIGNS* 



R - Random X - Treatment 

0 - Pretest ff - Posttest 



Design No. 1 
R X 
R X 0 2 
R X #3 



Design No. 2 
R 0 X 
'R 0 Cfj 

R 0 X 0 2 
R 0 0 2 

R 0 X 0 3 
R 0 0 3 



Design No. 3 Design No. 4 

R X l 0 R C X, 0 

R X 2 0 R C X 2 ft 

R C ft 



* Campbell and Stanley, 1963 
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